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Abstract: The success of several constraint-based modeling languages such 
as OPL. ZINC, or COMET, appeals for better software engineering practices, 
particularly in the testing phase. This paper introduces a testing framework en- 
abling automated test case generation for constraint programming. We propose 
a general framework of constraint program development which supposes that a 
first declarative and simple constraint model is available from the problem spec- 
ifications analysis. Then, this model is refined using classical techniques such as 
constraint reformulation, surrogate and global constraint addition, or symmetry- 
breaking to form an improved constraint model that must be thoroughly tested 
before being used to address real-sized problems. We think that most of the 
faults are introduced in this refinement step and propose a process which takes 
the first declarative model as an oracle for detecting non-conformities. We derive 
practical test purposes from this process to generate automatically test data that 
exhibit non-conformities. We implemented this approach in a new tool called 
CPTEST that was used to automatically detect non-conformities on two clas- 
sical benchmark programs, namely the Golomb rulers and the car-sequencing 
problem. 

Key-words: software testing, constraint programming, conformity relation, 
constraint negation 
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Vers le test des programmes a contraintes 

Resume : Tout processus de developpement logiciel effectue dans un cadre 
industriel inclut desormais une phase de test ou de verification formelle, y com- 
pris pour le developpement des programmes a contraintes. Notre travail vise a 
definir unc plateforme de test qui genere automatiqucment des donnees de test 
pour les programmes a contraintes. Cette nouvelle plateforme est egalement 
motivee par le developpement recent de plusieurs langagcs de modelisation de 
haut-nivcau tcls que OPL, ZINC, COMET, ou GECODE, qui ouvre la voie a 
des recherches orientees vers les aspects genie logiciel autour de la programma- 
tion par contraintes. Notre approchc repose sur certaines hypotheses quant 
au developpement et rafhncment dans un langage de PPC. II est usucl de 
demarrer a partir d'un modele simple et tres declaratif, une traduction fidclc 
de la specification du probleme, sans accorder d'intcrct a ses performances. Par 
la suite, ce modele est raffine par l'introduction de contraintes redondantes ou 
rcformulees, l'utilisation de structures de donnees optimisees, et de contraintes 
globales. Nous pensons que l'essentiel des fautes introduites est compris dans 
cette deuxieme etape. Dans ce rapport nous batissons une plateforme de test 
des programmes a contraintes qui etablit les regies de conformite entrc le modele 
initial declaratif et le programme optimise dedic a la resolution d'instances de 
grandc taillc. Nous avons implements cette approchc en un premier proto- 
type de test appele CPTEST. Cet outil permet de detecter automatiquement 
des non-conformitcs entre le modele de depart et le programme raffine. Nous 
presentant egalement a la fin une premiere validation experimentale sur deux 
problemes connus, les regies de Golomb et le probleme d'ordonnancement des 
vehicules (POV). 

Mots-cles : test logiciel, programmation par contraintes, relation de confor- 
mite, negation de contrainte 
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1 Introduction 

Constraint programs such as those written in modern Constraint Programming 
languages and platforms (e.g. OPlfl COMET0, ZINC@, CHOCC0, GECODE|f 
...), aim at solving industrial combinatorial problems that arise in optimization, 
planning, or scheduling. Recently, a new trend has emerged that propose also to 
use CP programs to address critical applications in e-Commerce [5], air-traffic 
control and management [3J |5] , and critical software development [TJ 0] . While 
constraint program debugging drew the attention of some researchers, few sup- 
ports in terms of software engineering and testing have been proposed to help 
verify critical constraint programs. Automatic debugging of constraints pro- 
grams has been an important topic of the OADymPPaQj project, that resulted 
in the definition of generic trace models [H [7J , the development of post-mortem 
trace analyzers, such as Codeine for Prolog, Morphine [7] for Mercury, ILOG 
Gentra4CP, or JPalm/JChoco. These models and tools help understand con- 
straint programs and contribute to their optimization and correction, but they 
are not dedicated to systematic fault detection. Indeed, functional fault detec- 
tion requires the definition of a reference (called an oracle in software testing) 
in order to check the conformity between an implementation and its reference. 
Automatic fault detection also requires the definition of test purpose to de- 
cide when to stop testing. Whereas conventional software development benefits 
from research advances in software verification (including static analysis, model 
checking or automated test data generation) , developers of constraint programs 
are still confined to perform systematic verification by hand. 

Automatic constraint program testing cannot be easily handled by existing 
testing approaches because of the two following reasons: firstly, constraint pro- 
grams are intrinsically non-deterministic as they represent sets of solutions and 
conventional definitions of conformity do not apply ; secondly, the refinement 
process of constraint programs is specific to CP. Indeed, developers usually start 
with an initial declarative constraint model of the problem, which faithfully 
translates the problem specifications, without granting interest to its perfor- 
mances. As this model cannot handle large-sized instances of the problem, they 
exploit several refinement techniques to build an improved model. For exam- 
ple, usual refinement techniques include the use of dedicated data structures, 
constraint reformulation, global constraints addition, redundant and surrogate 
constraint addition, as well as constraints which break symmetries (these con- 
straints usually improve considerably the effectiveness of the solving process). 
The refinement process, carried out by the developer, is an error-prone process 
and we believe that most of the faults are introduced during this step. 

In this article, we propose a testing framework for checking the correction 
of a constraint program implementation. The oracle for the constraint program 
under test is an initial declarative model considered to be valid w.r.t. the user 
requirements. Our framework is based on the definition of four distinct confor- 
mity relations to handle constraint satisfaction problems as well as optimization 



1 www. ilog.com/products/oplstudio/ 

2 www. dynadec.com/support/downloads/ 

3 http://www.gl2. cs.mu.oz.au/ 
4 choco. sourceforge.net 

5 www. gecode.org 

6 http://contraintes. inria.fr/OADymPPaC/ 
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problems. A practical consequence of these definitions is the proposal of test 
purposes for evaluating the conformance of constraint programs. Note that this 
paper does not address another essential topic of CP verification which is the 
correction of solvers or optimizers. We propose an algorithm for checking the 
correction of the CP program under test that solves a set of derived constraint 
problems able to exhibit non-conformities. We implemented our approach in a 
tool called CPTEST that seeks non-conformities in OPL programs. For evalu- 
ating the proposed testing process, CPTEST was used to find non-conformities 
in various faulty OPL constraint programs of the Golomb rulers and the car- 
sequencing problem. It was also used to assess the conformity for small instances 
of the problem. 

The rest of the paper is organized as follows: Sec. [5] illustrates our testing 
framework on a simple case in order to show a typical non-conformity case. 
Sec. [3] gives the definition of conformity relations required in the framework. In 
Sec. 21 the testing process we derive from these definitions is introduced and 
illustrated on a simple example. Sec. [5] presents the CPTEST tool and details 
our experimental evaluation. Finally, Sec. [5] concludes the paper and draws 
some perspectives to this work. 

2 An illustrative example 

Let us illustrate some of the refinement techniques on the classical problem 
of the Golomb rulers, which has various applications in fields such as Radio 
communications or X-Ray crystallography. 

A Golomb ruler [8] is a set of m marks = x\ < a; 2 < ... < x m such as 
m(m — l)/2 distances {xj — Xi\ 1 < i < j < m} are distinct. A ruler is of order 
m if it contains m marks, and it is of length x m . The goal is to find a ruler 
of order m with minimal length (minimize x m ). A declarative model of this 
problem is given in part A of Fig JT] while part B presents a refined and improved 
model. It is easy to convince a human that model A actually solves the Golomb 
rulers problem, but this is much more difficult for model B. Indeed, model B uses 
a matrix as data structure (d [indexes] ), statically breaks symmetries (cc6), 
it contains redundant and surrogate constraints (cc7,cc8,cc9) and global con- 
straints (allDif f erent). In this paper, we address the fundamental question of 
revealing non-conformities in between the constraint program under test B and 
the model-oracle A. Testing B before using it on large instances of the problem 
(when m > 15) is highly desirable as computing the global minimum of the 
problem for these instances may require computation time greater than a week. 
Note that B is syntactically correct and provides correct Golomb rulers for small 
values of m. Our testing framework tries to find an instantiation of the variables 
that satisfies the constraints of B and violates at least one constraint of A. This 
testing process is detailed in section @] With m = 8, our CPTEST framework 
computes x = [0 1 3 6 10 26 27 28] in less than 6sec on a standard machine, 
indicating that B does not conform A and then contains a fault. Indeed, x is 
not a Golomb ruler as 27 — 26 = 1 — = 1. In fact, this non-conformity can 
easily be tackled by removing the comment on constraint cc5 in part B. Doing 
so; CPTEST provides a conformity certificate saying that the CP program ac- 
tually computes the global minimum in 10034. 69sec (about 3hours). However, 
note that this certificate is only valid for m = 8. Note also that our framework 
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int m= . . . ; 

dvar int+ x [1 . 
minimize x[m] ; 
subject to { 
cl : forall (i 

x[i] < x[i+l] ; 

c2: forall (i.j.k.l 

(i < j kk k < 1 

(i 1= k || j 1= 



m]j 



in 1 . .m-1) 



in 1. 

kk 

1))) 



} 



x[j] - x[i] != x[l] - x[k]; 



int m= . . . ; 

dvar int x[l..m] in 0..m*m; 
tuple indexerTuple { int i; int j;} 
{indexerTuple} indexes={<i, j> I i, j in l..m: i < j>; 
dvar int d [indexes]; 
minimize x [m] ; 
subject to { 
ccl: forall (i in l..m-l) 

x[i] < x[i+l] ; 
cc2: forall (ind in indexes) 

d[ind] == x[ind.i]-x[ind. j] ; 
cc3: x[l]=0; 

cc4: x[m] >= (m * (m - 1)) / 2; 
// cc5: allDif f erent (all (ind in indexes ) d[ind]); 
cc6: x[2] <= x[m]-x[m-l]; 

cc7: forall (indl in indexes, ind2 in indexes, 
ind3 in indexes: (indl . i==ind2. i)&& 
( ind2 . j ==ind3 . j ) kk ( indl . j ==ind3 . j ) kk 
(indl.i<ind2.j < indl.j)) d[indl]==d[ind2] +d[ind3] ; 
cc8: f oralKindl , ind2,ind3, ind4 in indexes: 
( indl . i==ind2 . i) kk ( indl . j ==ind3 . j ) kk 
( ind2 . j ==ind4 . j ) kk ( ind3 . i==ind4 . i) kk ( indl . i<m-l) 
kk (3<indl . j <m+l) kk (2<ind2 . j <m) kk ( Kind3 . i<m-l) kk 

(indl.i < ind3.i < ind2.j < indl.j)) 
d [indl] ==d [ind2] +d [ind3] -d [ind4] ; 
cc9: foralKi in 2..m, j in 2..m, k in l..m : i < j) 
x[i]=x[i-l]+k => x[j] != x[j-l]+k; 
} 

- B - 



Figure 1: M x (k) and P x (k) of Golomb rulers problem in OPL. 

can handle non-conformities of the Golomb rulers where the global minimum 
requirement is relaxed in order to deal with larger instances (when m > 30). 

3 Testing constraint programs 

3.1 Notations 

In the rest of the paper, x denotes a vector of variables and (x\Xi) stands for 
substituting x by the valuation Xi. 

A constraint program includes a constraint model M x (k), 
which is a conjunction of constraints Ci{x) over variables x pa- 
rameterized by k, the parameters vector of the model. Note that 
x may depend on k. For the Golomb rulers, k is the order of the 
ruler while x represents the vector of marks. If k — 3 then one 
seeks for a ruler with 3 marks (e.g., x=[0 1 3]) while if k = 4 
one seeks for a ruler with 4 marks (e.g., x=[0 1 4 6] ). SolveQ 
is a generic procedure representing either the call to a constraint solver in the 
case of constraint satisfaction problem or the call to an optimization procedure. 
In this latter case, we note / the cost function (for the sake of clarity, / will 
be a minimization function but maximization problems can be tackled as well). 
We consider that k belongs to /C the set of possible values of the parameters for 
which M x (k) has at least one solution. sol(M x (k)) denotes the set of solutions 
of M x (k) and while Proj y (sol(M x (k))) expresses the projection of sol(M x {k)) 
on the set y when y C x. In optimization problems, one usually starts with 
feasible solutions ranging in a cost interval [l,u]. Therefore, we introduce the 



Model M x (k) 
(d(x) 

(C n (x) 
SolveQ 
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set 

Bounds f ,i,u(M x (k)) = {x\x e sol(M x (k))J(x) e [l,u]} 

To clarify these notations, Fig. [2] shows an example of a real objective function 
where point x\ is a global minimum with a cost f{x\) = b and points xo,xs 
belongs to Bounds f : i. u (M x (k)). Note that x\ as well as 22 do not necessarily 
belong to Bounds f,i, u (M x (k)) . 

3.2 Constraint models and programs 

In our framework, we consider the initial declara- 
tive constraint model to be a testing oracle, called 
the Model - Oracle , and noted M x (k). M x (k) rep- 
resents all the solutions of the problem and strictly 
conforms the problem specifications. We suppose 
that, for any parameter instantiation, M x (k) pos- 
sesses at least one solution. Considering unsatisfi- 
able Model-Oracles could be interesting for some 

applications (such as software verification [4]) but Figure 2: Objective solutions, 
we excluded these cases in order to avoid consid- 
ering equivalence of unsatisfiable models. The Constraint Program Under Test 
(CPUT) is a constraint model P z (k) (possibly unsatisfiable) which has to be 
tested for correction against the Model-Oracle. P z (k) is intented to solve diffi- 
cult instances of the problem. We built our framework on the hypothesis that 
checking whether M( x \ Xo )(fco) is true where xq is a point of the search space is 
not hard, while finding such an xq satisfying the constraints may be hard. Given 
a CPUT P z (k) and its Model-Oracle M x (k), we suppose that iCzas Pz(k) 
was obtained by refining M x (k). Hence, the set of variables in z distinct of x are 
dependant variables that are automatically instantiated when x is instantiated. 

3.3 Conformity relations 

The correction of a CPUT w.r.t. a Model-Oracle can be approached through 
the usage of conformity relations. These relations aim at assessing the correc- 
tion of the CPUT, a notion that can be expressed with various levels of depth. 
We propose four set-based definition of conformity divided on two groups: con- 
formity relations adapted to constraint satisfaction problems and conformity 
relations for optimization problems. 

3.3.1 Conformity relations for constraint satisfaction problems 

The simplest definition of correction, well-adapted for problems where a single 
solution is sought, is given by the following conformity relation: 

Definition 1 (conf one ) 

P conj k one M <*> Pro Jx (sol(P z (k))) ± A Pro Jx (sol(P z (k))) C sol(M x (k)) 
P conf one M *► (Vfc eK,P cmfte M) 

Roughly speaking, for a given instance k, conf^ ne asks the solutions of the 
CPUT to be included in the solutions of the Model-Oracle. As an example, 
Fig J3] presents both the sets sol(M x (k)) noted M and sol x (P z (k)) noted P, where 
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points in red x raise non-conformities (i.e., faults in the CPUT) while points in 
green o are conform w.r.t. the Model-Oracle. Parts (a)(b)(c) of FigJ3] exhibit 
non-conformities as solving P z (k) can lead to solutions which do not satisfy 
M x (k). Part (d) does not exhibit any non-conformity but, as P does not contain 
any solution, it does not conform the Model-Oracle for conf one . This example 
also shows that unsatishable models must be considered as non-conform w.r.t. 
Model-Oracles, in order to tackle faulty unsatishable CPUTs. On the contrary, 
part (e) of Fig[3] shows that P z (k) conforms M x (k) for conf one , as P cannot 
contain any non-conformity points. 



(a) 



non-conform 





(d) 




Fault 
Conformity 



Figure 3: conf one on P z (k) and M x (k). 

Whenever all the solutions are sought, another definition of conformity is 
useful: 

Definition 2 (conf a ii) 

P amf all M & Pro Jx ( S ol(P z (k))) = sol(M x (k)) (± 0) 
P conf all M <S> (Vfc elC,P conj k all M) 

Roughly speaking, conf a u asks for both set of solutions to be the same. Satis- 
fying this conformity relation is very demanding and not always pertinent. For 
instance, the CPUT in part B of FigfTJincludcs constraints that break symmetries 
of the problem (e.g., cc6), which yields to lose solutions from the Model-Oracle. 
As a result, those two models cannot be conform w.r.t. conf a u. In Fig. 31 



(a) 







x Fault 

o Conformity 



Figure 4: conf a u on P z {k) and M x (k). 



parts (a)(b)(c) and (d) exhibit non-conformities. Part (d) shows a solution of 
the Model-Oracle which is not solution of the CPUT ; therefore, the CPUT is 
a faulty over-constrained model. Part (c) exhibits the opposite case where the 
CPUT is a faulty under-constrained model. Proving that P z (k) conforms M x (k) 
for one of these two conformity relations is highly desirable. Unfortunately, such 
a proof would require not only to find all the solutions of the CPUT which is an 
NP_hard problem for some constraint languages (e.g., the finite domains con- 
straint language), but also to perform this for any value of k. This seems to be 
intractable in general (probably undecidable) and then we will confine ourselves 
to the search of non-conformities within finite resources. 
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3.3.2 Conformity relations for optimization problems 

Conformity relations for optimization problems is harder to define, as practicians 
usually start their refinement process by the definition of bounds for the optimal 
case [5] . Note also that non-conformities may arise in the cost function itself 
and we wanted our conformity relations to be able to tackle those cases. 





(c) 

Bp 



<£> 




Feasible solution 
Fault 



Figure 5: conf bounds on P x (k) and M x (k). 

FigJS] presents the conformity relation where feasible solutions of the CPUT 
are sought in [l,u\. Bp denotes the set Bounds f t i, u (P x (k)), Bm denotes the 
set Bounds f t i tU (M x (k)) while B is the set of global minima of M x {k). Part 
(a) exhibits four non-conformities as these points are not feasible solutions of 
the Model-Oracle M x (k) in [l,u]. For the same reason, Part (b) exhibits two 
non-conformities as two feasible solutions of Bp with cost in [/, u] do not belong 
to Bm- Part (c) presents also a non-conformity as Bp does not contain any 
feasible point meaning that the minimization problem cannot find a feasible 
solution with cost in [l,u]. On the contrary, part (d) shows conformity because 
solutions of Bp belong to Bm ■ Formaly speaking, 

Definition 3 (confbounds) 



P COn fbounds 



M <=> Proj x {boundsf,u u (P z (k))) ^ 

A Proj x (boundsfi tLu (P z (k))) C bounds f,i,u(M x (k)) 



Note that the definition of confbounds does not require that f = f and then cases 
where the cost function has been refined can also be handled. This conformity 
relation is useful for addressing hard optimization problems as it does not require 
the computation of global minima. As a result, it can be used to assess the 
correction of models on relaxed instances of the global optimization problems. 
We will come back on this advantage in the experimental validation section. 
However, for some problems, it may be useful to assess not only the correction 
but also the fact that the CPUT actually computes optimal solutions. This can 
be performed by using the following definition which ensures that the global 
optimum belongs to [l,u]. 



Definition 4 (confb est ) 



P confLt M <* 



(P™nf b k ounds M, 

< bounds f-oo,i{M x (k)) = 

[bounds p -oo.i(Pz(k)) = 



4 A CP testing framework 



Testing a CPUT w.r.t. an model-oracle requires to select test data. In this 
context, a test datum defines an instance of the CPUT and a point of the 
search space. 
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Definition 5 (Test datum) Given a CPUT P z {k) and a Model-Oracle M x (k), 
a test datum is an instantiated pair (ko,Xo) of parameters and variables. 

Note that evaluating Mk(x) on the test datum (ko,xo) results true when Xq 
is a solution of the model and false otherwise. Test execution is realized by 
evaluating both P z \ Za (fco) and M x \ Xo (fc Xj and checks whether the results (either 
true or false) are the same. Depending on the selected conformity relation, a 
test verdict can be issued. This elementary process can be repeated as long as 
one wishes, but it is more interesting to guide the test data generation process 
by the use of test purposes. Seeking non-conformities implies finding test data 
such as the CPUT is satisfied and the Model-Oracle is violated. This enables 
to detect faults in CPUT, and helps the constraint programmer to revisit its 
refinements. Based on the selection of a conformity relation, non-conformities 
can be sought with the following test purposes: 

conf one Given k, find a solution to P z (k) A —>Ci where Cj is a constraint of 
the Model-Oracle M x (k). The idea here is to isolate a non-conformity by 
looking independently at each constraint of the model-oracle. Considering 
all the constraints of the model-oracle would also be possible but less 
efficient to detect non-conformities as more constraints would be involved. 
Note that heuristics can be defined on the order of constraints to consider 
first. Note also that proving the unsatisfiability of P z {k) A -iCj for all Cj £ 
M x (k) permits to issue a conformity certificate saying that Pconfg ne M. 

confaii Given k, find a solution to (M x (k) A -"C-) V (P z (k) A ->Cj) where C 
(resp. Cj) is a constraint of the Model-Oracle M x (k) (resp. P z (k)). In 
this case, proving the unsatisfiability of these constraints permits to issue 
the conformity certificate Pconf^M, but this is not often desirable as 
constraint solving usually requires to issue a single solution instead of all 
solutions. 

confounds Given k and [l,u], find a solution to P z (k) A ->Ci A f'(z) € [l,u] A 
f{x) £ [I, u] where /, /' are the cost functions of the Model-Oracle M x (k) 
and the CPUT P z (k). Proving that these constraints are unsatisfiable 
permits to issue a certificate Pconf£ ounds M. 

confbest Given k, find a solution to (P~ [ Conf^ ounds M)\ 'bounds f^ 00> i(M x (k)) ^ 
V bounds f t -oo t i(P z (k)) ^ 0. Proving that these constraints are unsatis- 

Hesf 



liable permits to issue a conformity certificate Peon ft 



Interestingly, any solution found by the guidance of one of these test purposes 
can be stored for further investigations. Indeed, it can be used to debug the 
CPUT by looking at the violated constraint and it can also enrich a test set that 
will serve to assess the correction of future versions of the CPUT. In addition, 
conformity certificates are essential for those who want to convince third-party 
certification authorities that their CP programs can be used in critical systems 
[5J|3]. So, the proposed testing framework has a role to play in various phases 
of the constraint program development. 



7 zq is obtained by extending xq with values depending on xq 
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We now propose a simple but generic algorithm for searching non-conformities 
(Algorithm [TJ. 



Algorithm 1: one_negated(£>, {C\, ...C n }) 



Input : D, {Ci, ...C n } set of constraints. 

Output: conf when {Ci, ...C n } conform £>, -^conf{+ non-conformity point) otherwise 



nc <- V) 

X <— vars(D) 

foreach C; S {d . ..., C„} do 
V +- vars(Ci)/X 

if V = then nc <- Solve(D A ^d) 
else nc <— Solve(D A ^Projx(Ci)) 
if nc then return -icon/(nc) 

end 

return conf 



where Solve(D) denotes the algorithm to find the first solution of the constraints D 1 vars(D) 
denotes the set of variables in D and Projx(C) denotes the constraint projection on variables X. 

Algorithm [T] takes two constraint sets as input and returns either conf when 
both sets conform with relation conf one or ^conf (non-conformity point) where 
a non-conformity point has been found. Note that the other conformity re- 
lations can easily be implemented using this algorithm just by adjusting the 
call parameters. Special care has to be taken when building the negation of a 
model. For example, consider a Model-Oracle M with x-y ! =x-z ; x-y ! =y-z ; 
x-z!=y-z; and a CPUT P with cl: x-y=dl ; c2: x-z=d2; c3 : y-z=d3; 
c4: allDif f (dl,d2,d3) ;. Here, it is trivial to see that Pconf a iiM but if cl 
is selected for negation, M A -icl has solutions as dl is out of the scope of M. In 
the definitions of the conformity relations, these cases were discarded by the use 
of projections on the variables of the model-oracle. As computing general pro- 
jections is expensive, pragmatic solutions are available in our implementation 
(see SecJS]). 

Algorithm [T] is the core algorithm of the presented testing framework and 
several implementation improvements are described in Sec l5.ll Providing that 
the underlying constraint solver is sound and complete, this algorithm is sound 
as it cannot report conf if there exists a non-conformity point. Indeed, given 
k, upon completion of the algorithm the unsatisfiability of P z (k) A -iM x (k) is 
demonstrated showing that both models conform with the selection conformity 
relation. It is also complete as it cannot report false non-conformities. 

A kcypoint of our approach is that test data can be automatically gener- 
ated using the same constraint solver as the one used for solving the CPUT. 
Recall that we rely on the solver and we are only interested in detecting non- 
conformities in models. 

5 Experimental validation 

5.1 Implementation 

We implemented the testing framework shown above in a tool called CPTEST 
for OPL (Optimization Programming Language [TO])- We chose OPL because 
it is one of the main programming environments for developing constraint pro- 
grams and also critical constraint programs [2j- CPTEST is based on ILOG CP 
Optimizer 2.1 from ILOG OPL 6.1.1 Development Studio. All our experiments 
were performed on Quadcore IntclXcon 3.16Ghz machine with 16GB of RAM 
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Table 1: Syntax of OPL expressions handled by CPTEST 

Ctrs : := Ctr | Ctrs 

Ctr : := rel | foralK rel ) Ctrs\ or( rel ) Ctrs | if ( rel ) Ctrs else Ctrs 

allDif f erent (re/) | allMinDistance (re/) | inverse(re/) | f orbiddenAssignments (re/) 

aiiowedAssignments (re/) packCreO 



and all the models we used to perform these experiments are available online at 
www.irisa.fr/celtique/lazaar/CPTEST. 

CPTEST includes a complete OPL parser and a backend process that pro- 
duces dedicated OPL programs as output. These OPL programs must be solved 
in order to find non-conformities. If a solution is found, then CPTEST stops 
and reports the non-conformity to the user. Whenever all these OPL programs 
are shown to be inconsistent, then a conformity certificate is issued. The tool is 
parameterized by several options, including the chosen conformity relation, the 
instance of the problem, etc. CPTEST handles the overall OPL language and 
can negate most of the constraints that can be expressed in OPL. However, it 
cannot negate all the global constraints available, such as the cumulative or 
circuit global constraint. TabfT] summarizes the syntax of OPL constraints 
handled by CPTEST. OPL includes two aggregators, namely f orall and or. 
The universal qualifier f orall is used to declare a collection of closely related 
constraints and to build global constraints. Interestingly, the or aggregator can 
be used to negate f orall, as or implements existencial quantification. The 
OPL If-then-else statement is less general than it may appear as its con- 
dition cannot contain decision variables. Its negation can be computed by 
negating the Then-part and Else-part without any loss of generality, as our 
goal is only to find non-conformities instead of computing the negation of a 
general model. Our CPTEST tool handles several global constraints over dis- 
crete values, namely aiiDiff erent, allMinDistance, inverse, f orbiddenAssignments, 
aiiowedAssignments and pack. These constraints can be represented as an aggrega- 
tion of constraints and then computing their negation becomes trivial with the 
rules presented above and using the other global constraints. For example, the 
negation of C: allDiff erent (all (i in R) x[i]) is or (ordered i,j in R) 
x[i] = x[j] as C rewrites to for all (ordered i,j in R) x[i] != x[j],and 

the negation of f orbiddenAssignments is simply aiiowedAssignments. 

We implemented algorithm [1] in CPTEST with several improvements. In 
particular, by noticing that it is unnecessary to search for non-conformities 
on constraints that are included in both the CPUT and the Model-Oracle, we 
implemented a simple rewriting system to check equality modulo Associativity- 
Commutativity (=ac)- The system implements the following rules: 

X o y — > y o x, (x o y) o z — > x o (y o z), x + — > X, 

x * 1 — > x, x * — >• 0, x x (y • z) — > (x X y) • (x X z) , 

x < y O y > x, x < y <-> y > x, x — ^ x, 

where o e {+, *, A, V}, x e {*, A, V} and • 6 {+, A, V}. 

In algorithm [TJ the constraint Ci is discarded whenever there exists C'i in 
D such as C[ =ac (Cj)- 

We have seen in sec|5] that computing general projection is expensive, we can 
enumerate some practical solutions to handle local variables and the constraint 
projection: 
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Table 2: Faulty versions of the Golomb Ruler 





constraints of P present in the CPUT 


CPUT1 


ecl, cc9 




CPUT2 


ecl, cc2, cc7, cc9 




CPUT3 


ecl, cc2, cc7, cc8, cc9 




CPUT4 


ecl, cc2, cc3, cc4, cc6, cc7, cc8, cc9, cclO 



• Annotating constraints of CPUT. 

• computing projections (Fourier elimination). 

• checking non-conformities. 

It is important to stress that projections arc computing when we seek all 
solutions {con f a a) and we have d G P to negate (M A ->Cj). We implement 
in CPTEST the first and the last proposed solution. The CPTEST user's can 
annotate his CPUT by indicating the constraint that connects base and local 
variables. Otherwise, CPTEST check if the non-conformity reached is a real 
one or a false alarm. 

The goal of our experimental evaluation was to check that CPTEST is able to 
detect faults in OPL programs. We feeded CPTEST with faulty models coming 
from initial constraint program development. Indeed, we developed optimized 
models of two well-known CP problems, namely the Golomb rulers and the car 
sequencing problem, and we kept first versions of these models for which faults 
were found. 

5.2 The Golomb ruler problem 

The model-oracle of the Golomb rulers is given in part A of FigQ] while part 
B contains a conform version of an optimized version of the model when the 
comment on constraint cc5 is removed. Let us call P this version. The four 
intermediate versions of the Golomb rulers we kept from our initial program 
development contain realistic faults, not invented for the experiment. TabJ5] 
shows the four faulty versions expressed with the constraints of P. Note that 
constraint cc6 breaks symmetries in the problem and then it removes solu- 
tions (valid Golomb rulers) w.r.t. the model-oracle. Constraint cclO is not 
documented in P, it corresponds to forall( i in m. .3*m) count (all (j in 
indexes) d[j] ,i)==l. For each CPUT, we studied its conformity w.r.t. the 
model-oracle (part A) using the four conformity relations. The results we got 
for an instance parameter m = 8 are given in Tab|3l For the confb oun( is rela- 
tion, the interval [50, 100] was used to feed the relation, knowing that the global 
minimum is x m = 34 when m — 8. Each time a non-confirmity was found, it 
was reported with the CPU time required to find it. Firstly, the four faulty 
CPUT were reported as being non-conforms and the time required for finding 
these non-conformities is acceptable (less than a few minutes in the worst case). 
Secondly, this experiment shows that the most practical conformance relations 
(i.e., conf one and con f bounds) are preferable to the other ones for efficiency rea- 
son. Indeed, for the first three CPUT, these relations gave results less than 
lOsec. Note that non-conformities are represented either by invalid Golomb 

RR n° 7291 



On Testing Constraint Programs 



13 



Table 3: Non-conformities found by CPTEST in various CPUTs of the Golomb 
rulers problem (timeout = lh30). 



m = 8 


COTl J one 


conf a ii 


COTlJbounds 


COTlfbest 












Non-conf p 


lints 


[0 7 8 18 24 26 35 44] 


[17 18 20 25 34 45 49 55] 


[0 2 3 6 11 58 72 86] 


[0 1 3 6 10 15 24 33] 


CPUT1 


T(s) 


4.29s 


21.45s 


5.64s 


7.31s 












Non-conf p 


>ints 


[0 4 5 26 28 31 47 63] 


[17 18 20 25 34 45 49 55] 


[0 18 39 43 45 46 55 64] 


[0 3 4 9 13 15 24 33] 


CPUT2 


T(s) 


5.62s 


40.78s 


4.64s 


174.43s 












Non-conf p 


Dints 


[0 4 5 26 28 31 47 63] 


[0 4 5 26 28 31 47 63] 


[0 18 39 43 45 46 55 64] 


[0 3 4 9 13 15 24 33] 


CPUT3 


T(s) 


9.53s 


45.78s 


7.15s 


389.04s 












Non-conf p 


,ints 


[0 12 18 20 29 33 34 39] 


[1 2 10 22 33 55 57 60] 


[0 21 30 32 42 45 46 50] 


[0 6 13 21 22 25 27 32] 


CPUT4 


T(s) 


12.60s 


0.15s 


9.01s 


12.53s 


Non-conf p 


Dint, 


conf 


[0 7 9 12 37 54 58 64] 


conf 




P 


T(s) 


3 448.46s 


0.18s 


3 658.13s 


timeout 



rulers (e.g., 44 — 35 = 35 — 26 = 9 in the CPUTl/con/ orie case) or by valid 
Golomb rulers (e.g., CPUTl/con/ a /z case). In fact, a valid Golomb ruler r can 
be produced when the model-oracle is satisfied by r while the CPUT is refuted 
by r. These non-conformities correspond to cases where the CPUT misses solu- 
tions of the problem. Interestingly, P is shown as being non-conform with the 
conf An relation and the non-conformity that is found represent a valid Golomb 
ruler (i.e., [0 7 9 12 37 54 58 64]). In fact, recalling that P includes constraints 
that break the symmetries, this result was expected. Finally, note that confor- 
mity of P when conf best is selected was impossible to assess within the allocated 
time (timeout=lh30). In fact, computing the global minimum of the Golomb 
ruler rapidly becomes hard even for small values of to (e.g., CPUT3/con/f, es t). 
Our experimental evaluation also had the goal to check that computing non- 
conformities with CPTEST was less hard than computing solutions. For that, 
we compared the CPU time required to find non-conformities with conf one 
when the parameter value to increases and the time required to solve Golomb 
on these instances. Fig|6] shows that finding non-conformities with CPTEST 
remains tractable until to = 23 while solving the CPUT becomes intractable as 
°""" Qg m ~~> 1 Q 



runtime(s) 



seeking non-conformity {CPTEST) 



solve Golomb Rulers (OPL) 




10 11 12 13 14 15 16 17 IS 19 20 21 22 23 

Ruler order (m) 



Figure 6: Testing time and solving time comparison on the Golomb rulers. 
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Table 4: Non-conformities found by CPTEST in various CPUTs of the car 
sequencing problem (timeout = lh30). 







^ on Jone 


Conf a ii 




10 slots 


55 slots 


10 slots 


55 slots 


CPUT1 


Non-conf points 


4536465132 


pi 


4546365132 




T(s) 


0.30s 


1.23s 


2.49s 


timeout 


CPUT2 


Non-conf points 


4631523546 


P 2 


5435462631 


— 


T(s) 


0.85s 


1.65s 


1.20s 


timeout 


CPUT3 


Non-conf points 


5236143645 


P 3 


5435462631 


— 


TOO 


0.24s 


0.70s 


90.73s 


timeout 


CPUT4 


Non-conf points 


conf 


conf 


1362645345 


p4 


T(s) 


0.96s 


1.06s 


1.26s 


100.22s 


P 


Non-conf points 


conf 


— 


6453452631 


— 


T(s) 


3.01s 


timeout 


0.17s 


timeout 


pi =6564524443567633356455227342555413416431533616777263164 
P 2 = 7163461732517354266643653442461375525537631643542465543 
p3 = 4315655124236663252174443335436466417315642576355674375 
p4 =1362543526453452635445376413671763146752631764543546253 



5.3 The car sequencing problem 

The car sequencing problem (CScq) illustrates interesting features of CP in- 
cluding wide parameter settings, redundant, surrogate and global constraints 
addition, and specialized data structures definition. This is a constraint satis- 
faction problem that amounts to find an assignment of cars to the slots of a 
car-production company, which satisfies capacity constraints. 

As a model-oracle of this problem, we took the model given in the OPL book 
|10j . In this model, capacity constraints are formalized by using constraints r 
outof s, saying that from each sub-sequence of s cars, a unit can produce at 
most r cars with a given option. Starting from this model, we built an opti- 
mized model by introducing several refinements, including a new data structure 
setup [o,s] which takes value 1 if option o is installed on slot s, redundant and 
global constraint addition (e.g., pack constraint). When building our improved 
model of car sequencing, we recorded four faulty constraint models that are 
used for experiments. Here again, the idea was to keep models that represent 
realistic faults instead of a posteriori injected faults. These four models arc 
available online on the site mentioned above. 

TabQ] gives the results of CPTEST on two instances of the problem: an 
assembly line of 10 cars, 6 class and 5 options ; an assembly line with 55 cars, 7 
class and 5 options. Using conf one , CPTEST reports non-conformities for the 
three first CPUT in less than lsec for both instances. CPUT4 has no solution 
as the fault introduced on the pack constraint prunes dramatically the search 
space. This case is interesting as detecting this fault is really uneasy. With the 
conf a u relation, the results are balanced as three instances were not detected 
as non-conformant within the allocated time slot. For example, in CPUT2, 
the capacity constraint of the first option is violated (1 out of 2). This fault 
results from a bad formulation but it is quickly detected with conf one . When 
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confaii is selected, more constraints have to be negated and then our algorithm 
has to backtrack a lot, which explains the failure. The non-conformity reached 
in this case satisfies the model-oracle and violates CPUT2, so it represents a 
correct assembly line that CPUT2 excludes from its solutions. Therefore, we 
can conclude that CPUT2 adds and removes solutions which make it difficult 
to detect as non-conform. 

6 Conclusion 

In this paper, we introduced for the first time a testing framework that is 
adapted to standard CP development processes. The framework is built on 
solid notions such as conformity relations, oracles and test purposes that are 
specific to CP. We also presented CPTEST an implementation of our frame- 
work dedicated to the testing of OPL programs and evaluated it on difficult 
instances of two well-known constraint problems, namely the Golomb ruler and 
car-sequencing problem. Our experimental evaluation shows that CPTEST can 
efficiently detect non-trivial faults in faulty versions of those two problems. A 
desirable extension of our framework and tool concerns its application to other 
more open CP platcforms. In particular, we would like to apply our confor- 
mity relations, oracles and testing notions to GECODE or CHOCO programs 
as we could intervene on the core constraint solver of these systems. Develop- 
ing notions of test coverage similar of those that can be found in conventional 
programming requires instrumenting the solver, something that was just not 
possible with the black-box solver of OPL. 
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